fix(native-eval): rebootstrap replacement leases by vincentkoc · Pull Request #58 · openclaw/shellbench

vincentkoc · 2026-07-29T16:20:43Z

What does this PR do?

Rehydrates replacement native-eval leases before dispatch.

Why?

The controller previously treated an old bootstrap timestamp as proof that a
new lease contained the runner. Recovery could then fail before the harness
started, producing a false benchmark failure.

Fixes #55

Changes

record the exact bootstrapped_lease_id
reuse bootstrap state only when it matches the active lease
add a regression test for recovery with an old timestamp and replacement lease

Live proof

Recovered a legacy OpenClaw run whose manifest had an old bootstrap timestamp
but no bootstrapped_lease_id.

rebound the run to its active replacement lease
performed the full native-runner bootstrap on that lease
persisted the exact active lease as bootstrapped_lease_id
dispatched the OpenClaw harness successfully
exported a verified final artifact with 4/4 completed results
run exit code: 0

This is the exact recovery state the regression test covers; the controller no
longer trusts timestamp-only bootstrap state.

Tests

40 focused fleet tests pass
Python 3.11 and Python 3.12 CI pass
Ruff passes on the touched files
fresh Codex autoreview reports no actionable findings
live replacement-lease recovery completes a four-task OpenClaw run

clawsweeper · 2026-07-29T16:25:19Z

Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 5:05 AM ET / 09:05 UTC.

ClawSweeper review

What this changes

The PR ties native-evaluation bootstrap reuse to the active Crabbox lease ID and adds a regression test for recovery after a lease replacement.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

This member-authored PR remains necessary: current main still uses a timestamp alone to decide whether a recovered native-evaluation lease is bootstrapped, while the branch records and checks the actual lease identity. The targeted regression test covers the reported replacement-lease recovery path; no discrete patch defect was found, but the intended legacy-manifest rehydration behavior warrants normal maintainer review before merge.

Priority: P2
Reviewed head: 8949dc8328e9b86a0cf1f80bcf3a976c67c4837a

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused repair with source-aligned behavior and a precise regression test; it needs ordinary maintainer confirmation of the intentional legacy recovery behavior.
Proof confidence	🌊 off-meta tidepool	Not applicable: This is a member-authored PR, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live four-task recovery run with exit code 0.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This is a member-authored PR, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live four-task recovery run with exit code 0.
Evidence reviewed	5 items	Current-main behavior: Current main checks only `bootstrapped_at_utc` before dispatching a missing remote run, so a timestamp carried from an expired lease can suppress hydration of a replacement machine. Patch behavior: The branch reloads the persisted run after lease assignment, hydrates when `bootstrapped_lease_id` differs from the active lease, and persists that lease ID only after hydration succeeds. Regression coverage: The added test starts a recovery-required run with an old timestamp and replacement lease, verifies hydration occurred, verifies the active lease ID was persisted, and verifies dispatch completed.
Findings	None	None.
Security	None	None.

How this fits together

The native-evaluation fleet controller resumes benchmark runs from a persistent run index and dispatches them onto Crabbox leases. Before dispatching to a machine with no remote run state, it decides whether to hydrate the runner, task archive, and provider environment; this PR makes that decision lease-specific.

flowchart LR
  A[Persistent run index] --> B[Recovery controller]
  B --> C[Active Crabbox lease]
  C --> D[Remote run-state probe]
  D --> E{Bootstrap lease matches?}
  E -->|No| F[Hydrate runner and tasks]
  E -->|Yes| G[Reuse hydrated lease]
  F --> H[Dispatch benchmark harness]
  G --> H

Before merge

Resolve merge risk (P1) - On upgrade, a legacy manifest without bootstrapped_lease_id will rehydrate once whenever the remote run state is missing, even if its stored timestamp is present; that deliberately replaces the remote runner, public tasks, and provider environment before dispatch.
Complete next step (P2) - A member-authored patch with no mechanical defect needs a maintainer to confirm the intentional upgrade behavior before merging.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	32 added, 1 removed across 2 files	The implementation is limited to one recovery decision, one persisted field, and one focused regression test.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #55
Summary: This PR is the concrete candidate fix for the reported replacement-lease recovery defect tracked by the linked issue.

Members:

canonical: Rebootstrap native evals when recovery replaces a lease #55 - The issue describes the same timestamp-only bootstrap reuse failure, and the PR body explicitly connects this branch to that recovery behavior.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

Confirm the legacy recovery tradeoff (recommended)
Approve the intentional one-time rehydration of legacy manifests without a recorded lease ID before merging the targeted recovery repair.
Pause for a different upgrade policy
Keep the PR open only if maintainers want a compatibility mode that trusts legacy timestamps rather than conservatively rehydrating missing remote state.

Technical review

Best possible solution:

Adopt lease identity as the durable bootstrap invariant, preserving the conservative one-time hydration path for legacy manifests and the existing successful recovery behavior for leases that already match.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: a recovery_required entry with an old bootstrapped_at_utc, no recorded bootstrap lease ID, and a replacement active lease follows the affected dispatch path. The branch adds a focused regression test for that exact state, but this read-only review did not execute it.

Is this the best way to solve the issue?

Yes. Recording the identity of the lease that actually received bootstrap is the narrowest durable fix; a timestamp cannot establish that a replacement machine has the required runner and task contents.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

P2: This is a bounded native-evaluation recovery defect that can produce false benchmark failures after a replacement lease.
merge-risk: 🚨 compatibility: The persisted bootstrap decision changes for existing manifests that lack a lease ID, causing a conservative rehydration on recovery.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This is a member-authored PR, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live four-task recovery run with exit code 0.

Evidence

What I checked:

Current-main behavior: Current main checks only bootstrapped_at_utc before dispatching a missing remote run, so a timestamp carried from an expired lease can suppress hydration of a replacement machine. (scripts/native_eval/fleet.py:566, 884dd1bb5511)
Patch behavior: The branch reloads the persisted run after lease assignment, hydrates when bootstrapped_lease_id differs from the active lease, and persists that lease ID only after hydration succeeds. (scripts/native_eval/fleet.py:566, 8949dc8328e9)
Regression coverage: The added test starts a recovery-required run with an old timestamp and replacement lease, verifies hydration occurred, verifies the active lease ID was persisted, and verifies dispatch completed. (tests/test_native_eval_fleet.py:1276, 8949dc8328e9)
Feature history: The native fleet controller appears to have been introduced and subsequently stabilized by Vincent Koc in the available current-main history; this PR is a narrow follow-up on that path. (scripts/native_eval/fleet.py:541, b9acd9f7a010)
Current-main comparison: The PR head is not an ancestor of current main, and current main has no later commits on either touched path after this branch point; the lease-ID safeguard is therefore not already implemented there. (scripts/native_eval/fleet.py:566, 8949dc8328e9)

Likely related people:

vincentkoc: Available history shows Vincent Koc introduced the native matrix runner, stabilized its fleet behavior, and authored this focused recovery fix. (role: native-evaluation feature owner and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 8949dc8328e9; files: scripts/native_eval/fleet.py, tests/test_native_eval_fleet.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

Confirm that conservative rehydration of legacy manifests without a recorded lease ID is the desired upgrade behavior.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (21 earlier review cycles; latest 8 shown)

reviewed 2026-08-01T22:59:14.951Z sha 8949dc8 :: needs maintainer review before merge. :: none
reviewed 2026-08-02T13:17:00.915Z sha 8949dc8 :: needs maintainer review before merge. :: none
reviewed 2026-08-02T19:14:34.796Z sha 8949dc8 :: needs maintainer review before merge. :: none
reviewed 2026-08-02T20:37:22.956Z sha 8949dc8 :: needs maintainer review before merge. :: none
reviewed 2026-08-02T22:18:20.095Z sha 8949dc8 :: needs maintainer review before merge. :: none
reviewed 2026-08-03T00:19:32.496Z sha 8949dc8 :: needs maintainer review before merge. :: none
reviewed 2026-08-03T01:48:36.800Z sha 8949dc8 :: needs maintainer review before merge. :: none
reviewed 2026-08-03T04:09:32.245Z sha 8949dc8 :: needs maintainer review before merge. :: none

fix(native-eval): rebootstrap replacement leases

8949dc8

clawsweeper Bot mentioned this pull request Jul 29, 2026

Rebootstrap native evals when recovery replaces a lease #55

Open

vincentkoc marked this pull request as ready for review July 29, 2026 17:29

vincentkoc requested a review from a team as a code owner July 29, 2026 17:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(native-eval): rebootstrap replacement leases - #58

fix(native-eval): rebootstrap replacement leases#58
vincentkoc wants to merge 1 commit into
mainfrom
fix/native-fleet-rebootstrap

vincentkoc commented Jul 29, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 29, 2026 •

edited

Loading

Security

Review metrics

Root-cause cluster

Merge-risk options

Technical review

Labels

Evidence

Rank-up moves

Rating scale

Workflow

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

vincentkoc commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why?

Changes

Live proof

Tests

Uh oh!

clawsweeper Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ClawSweeper review

What this changes

Merge readiness

Review scores

Verification

How this fits together

Before merge

Security

Review metrics

Root-cause cluster

Merge-risk options

Technical review

Labels

Evidence

Rank-up moves

Rating scale

Workflow

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vincentkoc commented Jul 29, 2026 •

edited

Loading

clawsweeper Bot commented Jul 29, 2026 •

edited

Loading